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SUMMARY 


Combat  Active  Replacement  Factors,  or  CARFs,  are 
logistics  planning  factors  currently  used  by  the  U.S.  Marine 
Corps  as  estimates  of  equipment  losses  in  future  conflicts. 
Adapted  Army  replacement  factors  are  currently  a  prime 
source  for  CARF  values,  but  verification  of  these  values 
is  difficult. 

This  report  examines  two  alternate  means  of  estimating 
CARF  values  for  comparative  purposes.  The  first  employs 
mean-time-to-loss  estimates  for  various  equipment  types, 
and  several  scenario-oriented  models  are  given  for  mapping 
these  values  into  CARFs. 

Professional  military  judgement  provides  another  way 
of  estimating  CARF  values,  and  a  procedure  is  given  by 
which  the  judgement  of  many  experts  can  be  aggregated 
to  provide  CARF  information.  A  demonstration  of  the 
procedure  is  given,  employing  actual  data  from  twenty- 
three  judges  who  rated  "chance  of  loss"  for  twenty-one 
equipment  types . 
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I.  INTRODUCTION 


Combat  Active  Replacement  Factors,  or  CARFs,  are 
logistics  planning  factors  currently  used  by  the  U.S. 

Marine  Corps.  As  estimates  of  equipment  losses  in  future 
conflicts,  their  values  have  a  significant  impact  on 
procurement,  stockpiling,  and  plans  for  shipping  requirements 
Concern  over  the  accuracy  of  CARF  values  led  to  the 
establishment  by  the  Marine  Corps  in  1983  of  a  standing  board 
for  the  review  of  materiel  replacement  factors.  The  board's 
specified  functions  include  the  review  of  "  .  .  .  replace¬ 
ment  factors  for  new  materiel  entering  the  Marine  Corps 
Supply  System,  existing  replacement  factors  for  materiel 
presently  in  the  Marine  Corps  Supply  System,  and  other 
services'  replacement  factors  .  .  .  where  such  factors 
would  prove  useful  in  confirming  the  validity  of  common 
item  factors."^  The  board's  task  is  a  difficult  one:  the 
current  Marine  Corps  primary  source  of  CARF  values  is  that 
of  adapting  Ainny  replacement  factors  for  like  items,  but 
the  Marine  Corps  lacks  available  war  gaming  and  simulation 
resources  to  permit  careful  verification  of  replacement 
factors  by  these  means.  This  report  discusses  some  ways 
of  generating  comparative  CARF  values  (for  verification 
use)  using  procedures  which  incorporate  military  judgement 
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and  do  not  require  computerized  war  games. 

Combat  Active  Replacement  Factors  (CARFs) 

I 

The  War  Reserve  Policy  Manual  of  the  Marine  Corps 
states  that  a  replacement  factor  is  .  .  the  estimated 
percentage  of  equipment  in  use  that  will  require  replacement 
during  a  given  period  due  to  wear-out  beyond  repair,  enemy 
action,  abandonment,  pilferage,  and  other  causes,  except 
catastrophies.  .  .  .  The  Marine  Corps  expresses  replacement 
factors  as  quantities  required  for  a  30-day  period." 

Further,  "The  combat  active  replacement  factor  will  be 
applied  for  units  during  those  periods  when  they  are  actually 
in  active  combat  operations.  A  force  in  contact  with  the 
enemy  is  considered  to  be  active  combat.  The  combat  active 
replacement  factor  reflects  anticipated  combat  attrition 
of  equipment,  on  a  30-day  basis,  incident  to  amphibious 
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operations  and  other  combat  operations  normal  to  the  FMF". 
Numerical  values  for  CARFs  for  a  wide  variety  of  equipment 
items  are  listed  in  the  Marine  Corps  Table  of  Authorized 
Materiel . ^ 

With  Marine  Corps  support,  several  studies  were  con¬ 
ducted  in  the  early  1980s  in  an  effort  to  develop  a  procedure 
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to  generate  CARF  values.  '  These  led  to  a  decision  to  use 
adapted  Army  replacement  factor  values.®  In  a  related  effort 
a  CARF  Determination  System  Users  Manual  was  prepared.^ 
Concern  over  verification  has  remained,  and  in  1985  a  number 


of  existing  war  games  were  surveyed  with  regard  to  their 
usefulness  to  the  Marine  Corps  for  CARF  generation  and 
verification.®  , 


This  Report 

This  report  looks  at  several  ways  of  employing  military 
judgement  to  estimate  CARFs  for  verification  purposes. 
Although  CARF  values  could  be  subjectively  estimated  dir¬ 
ectly,  one  alternative  is  to  employ  estimates  of  the  average 
combat  survival  time  of  an  item,  and  in  the  next  section 
we  will  look  at  ways  in  which  Mean  Time  Until  Loss  (MTTL) 
values  may  be  converted  into  CARF  values. 

Another  way  to  acquire  professional  military  judgement 
about  CARF  values  is  to  ask  that  various  equipment  items 
simply  be  ranked  with  regard  to  their  CARF  values. 

Section  III  of  this  report  explains  how  such  rankings  by 
many  judges  may  be  combined  to  yield  CARF  values  for  the 
various  items.  A  demonstration  of  this  method  is  then 
given,  scaling  chance  of  loss  for  21  functional  areas 
of  equipment.  The  report  concludes  with  appendices 
providing  some  cautions  regarding  CARF  generation  and 
use,  and  the  mathematical  derivation  of  the  scaling  method 
which  uses  rankings. 


:il.  CARF  GEljlERATION  AND  VERIFICATION 
USING  MTTL  ESTIMATES 

There  are  alternatives  to  the  use  of  computerized  war 
games  and  simulation  to  obtain  values  for  CARFs.  It  is 
possible  to  produce  estimates  directly,  perhaps  using 
professional  military  judgement  and  experience.  In  some 
cases  CARF  value  ,  so  generated  may  be  preferred  to  those 
obtained  from  coitibat  models,  since  there  may  be  more  clarity 
about  what  the  number  was  based  upon,  and  what  considerations 
went  into  its  estimation. 

A  variation  on  estimating  CARFs  directly  is  to  use 
professional  judgement  to  estimate  a  related  measure,  the 
mean  time  until  loss,  or  MTTL.  The  MTTL  is  the  average 
time  one  would  expect  an  item  to  survive  in  the  combat 
environment  (the  analogy  to  MTBF  is  appropriate) .  For  some 
items  and  some  individuals  asked  for  judgements,  the  MTTL 
may  be  a  more  natural  value  to  estimate  directly  than  a 
replacement  factor. 

Given  an  estimate  of  the  item's  MTTL,  however,  the 
relationship  between  MTTL  and  CARF  is  not  direct.  What  is 
needed  are  procedures  by  which  an  MTTL  estimate  for  an  item 
could  be  used  to  produce  a  CARF  value  for  the  item.  Such 
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procedures  necessarily  depend  upon  the  process  by  which  the 
losses  occur.  We  shall  look  at  several  kinds  of  scenarios 

by  which  losses  could  occur  in  combat,  and  then  develop 

« 

relationships  permitting  a  CARF  value  to  be  obtained  from 
an  MTTL  value.  These  loss  processes  include,  for  a  specific 
item: 

1.  Cases  where  all  items  of  that  type  are  vulnerable 
(subject  to  possible  loss)  initially  and  throughout 
the  combat  period,  at  the  same  rate. 

2.  Cases  where  subsets  of  the  items  in  use  have 
different  loss  rates. 

3.  Cases  where  the  rate  of  loss  changes  at  some 
designated  time  during  the  combat  period.  This 
permits  representation  of  an  amphibious  phase, 
or  a  force  buildup. 

4.  Cases  where  one  item  is  vulnerable  initially  and 
throughout  the  combat  period  at  the  same  rate,  but 
its  replacements  are  not  vulnerable  until  put  into 
use. 

Development  of  MTTL-CARF  relationships  for  these 
loss  processes  involves  characterization  of  the  probabilistic 
factors  by  which  the  loss  of  an  item  occurs. 


Loss  Process  Characteristics 

We  will  look  at  a  loss  process  where  the  event  that  an 
item  is  lost  is  independent  of  the  loss  of  any  other  on-line 
item  of  the  same  kind,  and  where  the  chance  of  a  surviving 
item  being  lost  on  the  next  day  is  independent  of  how  many 
days  it  has  already  been  in  combat.  This  means  that  the 
individual  item's  loss  rate  may  be  considered  constant 
over  the  period  of  time  we  are  examining. 


For  constant  loss  rates  and  independent  losses,  the 
probability  distribution  for  the  time  an  item  survives  in 
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combat  will  be  the  exponential  distribution.  Let  t  be 
the  combat  survival  time  for  a  specific  item,  and  let  R 
be  that  item's  loss  rate.  (The  units  of  R  are  items  per 
day.)  The  density  function  for  the  time  t  until  the  item 
is  lost  is 

f(t)  =  Re"^^  ,  t>0,  R>0  . 

From  this  we  may  immediately  obtain  the  probability  that 
the  item  is  lost  on  or  before  time  T,  F(T),  as 

FCT)  =>  Pr(t<T)  =  1  -  e""^^  . 

We  also  have  1  -  F(T),  the  probability  that  the  item 
is  still  surviving  just  after  time  T,  as 

1  -  FCT)  =  Pr(t>T)  s  e"'^^  . 

Since  the  expected  value  or  average  value  of  an 
exponentially  distributed  random  variable  is  the  reciprocal 
of  its  parameter,  we  now  have  the  mean  time  until  loss 
MTTL  as 

MTTL  =  1/R 

Accordingly,  the  chance  that  the  item  is  not  lost  during 
30  days  of  combat  is 

Pr{Not  lost  during  first  30  days)  =  e”^^/MTTL^ 
or  more  generally, 

Pr(Not  lost  during  first  D  days)  =  e”^/MTTL^ 


Loss  Rate  for  30  Days 

We  look  first  at  the  case  where  all  items  are  committed 
and  vulnerable  at  the  same  loss  rate  throughout  the  30-day 
period.  The  chance  that  an  item  is  lost  during  the  30  days 


,  ^-30/MTTL 

If  n  items  are  the  initial  in-use  amount,  committed  with 
independent  losses,  then  the  number  of  items  x  that  would 
be  lost  during  the  30-day  period  will  be  binomially  dist¬ 


ributed,  and  we  have 


X  out  of  n 


lost  in  30  days 


nl  ,,  _-30/MTTL,x,_-30/MTTL, n-x 

rrr^^U-e  j  le  j  , 


X  =  0,1,  . . . ,  n  . 


The  average  number  (out  of  n)  lost  in  30  days  is  simply  the 
mean  of  the  binomial  distribution,  or 
Ave  number  lost  =  n(l  -  e 

From  this  we  may  readily  obtain  a  CARF  value: 

(Ave  number  lost  in  30  days) (100) 


(in-use  amount) 


CARF  =  (1-e  30/MTTL, 
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Equation  (3)  permits  a  CARF  u6  be  computed  from  an  MTTL 
estimate  for  the  case  where  each  item  in  the  in-use  amount 
is  committed  initially  and  at  the  same  loss  rate,  and  where 

4 

losses  are  independent. 

Another  use  of  (3)  in  the  verification  of  an  existing 
CARF  value  would  ask  if  the  existing  CARF  value  would  yield 
an  MTTL  that  seemed  reasonable.  Here,  we  could  use 


MTTL  =  -30/ln{l-CARF/100)  .  (4) 


The  association  between  CARF  and  MTTL  for  this  case 
is  illustrated  numerically  by  the  values  in  Table  1. 


TABLE  1.  Some  Associated  CARF 
and  MTTL  Values  for  the  Case  of 
All  Items  Vulnerable  Initially, 
and  at  the  Same  Loss  Rate  for 
30  Days. 


MTTL:  Ave.  Number 

of  Days  Before  Loss  CARF,  % 

20  days  78  % 

30  63 

40  53 

50  45 

100  26 

200  14 

350  8 

500  6 


CARFs  Where  Not  All  Items  Have  the  Same  Loss  Rate 


If  all  items  are  init^lly  vulnerable,  but  with 
different  (constant)  loss  rates  for  the  30  day  period, 
then  CARF  generation  from  the  MTTL  values  is  an  extension 
of  the  case  where  all  items  have  the  same  loss  rate. 


For  a  general  formula,  let  proportions  Pj^,P2,  •••  /  Pj^ 
of  the  in-use  amount  n  have  mean  times  until  loss 
MTTLj^,  MTTL2/  •••  /  The  average  number  lost  in 

30  days  for  any  subgroup  i  is 


P.nd  -  , 


and  we  have 


As  an  example  of  the  use  of  (5) ,  suppose  half  of  the 
in-use  amount  have  MTTL  of  200  days,  one-quarter  have  MTTL 
of  100  days,  and  the  remaining  quarter  have  MTTL  of  30  days 
Then  from  (5) , 

CARF  =  100(Js(l  -  +  Js(l  -  +  hd  - 


=  100(15(0.14)  +  >*(0.26)  +  ls(0.63))  =  29.25  % 


This  is,  of  course,  simply  a  weighted  average  of  CARF 


values  from  Table  1. 
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CARFs  Where  All  Items  are  Vulnerable  Initially  and  There 
is  a  Change  in  the' Loss  Rate  During  the  30-Day  Period 

4 

It  may  sometimes  be  of  interest  to  construct  a  CARF 
for  a  situation  where  there  is  a  change  in  the  combat 
scenario  during  the  30-day  period.  One  immediate  example 
of  this  is  the  case  where  the  first  portion  of  the  30  days 
is  an  amphibious  operation,  and  the  loss  rate  might 
subsequently  change.  For  this  type  of  situation,  we  let 
1/MTTLj^  be  the  loss  rate  for  the  first  D  days  where  x^^ 
items  are  lost,  and  I/MTTL2  be  the  loss  rate  for  the 
remaining  (30  -  D)  days  where  X2  items  are  lost.  Here, 
total  losses  for  the  30-day  period  are  x^+  X2>  but  the 
problem  is  complicated  by  the  fact  that  the  initial  in-use 
amount  for  the  second  period  is  n  -  x^,  i.e.,  it  depends 
upon  losses  during  the  first  period. 

In  particular,  we  have  the  average  total  losses  for 
the  30-day  period  as 

n 

e[xi+X2I  =  e[xJ  +  ^e[x2|x^]  Pr(x^)  .  (6) 

xi=0 

Substituting  (2)  and  (3)  into  (6)  yields 

n  , 

e[xi+X2]  =  e[xi]  +  yjn-x^) 

(l-e-D/MTTLj,x^ ,^-D/MTTL^, n-x^ 
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This  simply  gives  us 


e[x^+X2]  =  n  [l  -  +  OO-Dj/MTTL^l 


The  CARF  would  be 


CART  =  (l  -  (30-D)/MTTL2) 


More  Complex  Scenarios 

The  methods  of  the  two  preceding  sections  may  be  applied 
for  the  purpose  of  generating  CARFS  in  other,  more  complex 
scenarios.  A  single  example  would  be  that  of  a  reserve  force 
committed  at  some  time  during  the  30  days.  The  total  initial 
number  of  items  might  be  divided  between  an  on-line  force 
and  a  reserve  force,  with  different  MTTL  values.  Commitment 
of  the  reserve  at  some  time  D  during  the  30-day  period  would 
decrease  its  MTTL,  and  possibly  increase  the  MTTL  of  the 
initial  on-line  force.  Equations  developed  in  the  preceding 
sections  offer  the  tools  to  construct  CARFs  for  this 
scenario,  and  for  others  with  more  complexity. 


CARFS  Where  an  On-line  Item  is  Replaced  by  a  Heretofore 
Invulnerable  Item 


The  expressions  for  generating  CARFs  in  the  previous 
sections  have  all  been  based  upon  a  situation  where  all 
items  were  initially  vulnerable  (although  loss  rates 
could  have  been  zero) .  A  different  loss  process  which  we 
now  examine  would  place  one  item  on  line,  and  structure 
its  (possibly  repeated)  replacement  with  heretofore 
invulnerable  items. 

If  the  supply  of  reserve  items  is  very  large,  or  large 
enough  so  that  the  chance  of  its  exhaustion  is  negligible, 
we  can  structure  this  case  by  noting  that  the  loss  process 
is  simply  a  sequence  of  exponentially  distributed  time 
intervals  over  a  30  day  period.  If  we  ignore  boundary 
conditions,  the  nximber  of  such  intervals  would  be  a 
Poisson  distributed  random  variable  with  a  mean  of 
30/MTTL.^  The  CARF  would  be 


CARF 


30/MTTL 
in-use  amount 


f 

100 

i 

(8) 


For  CARF  verification  we  could  ask  if 


(30) (100) 

MTTL  =  -  (9) 

(CARF) (in-use  amount) 


was  a  reasonable  number. 
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Equations  (8)  and  (9)  may  be  a  reasonable  approximation 
for  an  item  with  a  low  CARF  value-.  If  the  chance  of  running 
out  of  replacements  is  not  negligible,  however,  then (8)  will 
yield  an  overstated  CARF  value.  We -can  correct  for  this 
(at  the  cost  of  simplicity)  as  follows.  Let  n  be  the  initial 
amount  of  an  item,  and  x  be  the  losses  in  30  days.  Then, 
with  the  Poisson  probability  distribution  we  can  write 


x! 


,  x=0 , 1 , . . . , n-1 


f  (x)  =  < 


Z 


X  =  n 


x=n 


as  the  loss  distribution.  Taking  expected  values,  this 
yields  a  CARF  expression 


(9) 


(10) 


which  is  easily  programmed  for  numerical  solutions. 
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Remarks 

The  methods  given  in  this  section  provide  a  way  of 
generating  CARF  values  without  resort  to  war  games  or  to 
simulation.  Basic  to  this  method  is  the  estimation  of 
values  for  MTTL,  the  mean  time  until  a  loss,  and  the  method 
will  be  most  applicable  to  situations  where  the  MTTL  can  be 
estimated  with  more  confidence  than  the  needed  CARF  can 
be  estimated. 

The  process  assimptions  leading  to  the  use  of  the 
exponential  distribution  for  time  until  loss  are  analogous 
to  the  constant  failure  rate  assumption  which  is  so  widely 
used  in  reliability  theory. In  the  reliability  context 
this  assumption  is  widely  discussed  in  that  it  portrays 
chance  failures,  but  not  those  stemming  from  increased  age 
of  the  item,  or  wearout.  Constant  failure  rates,  however, 
are  widely  used  and  are  essentially  mandated  by  DOD 
needs  for  MTBF,  rather  than  mission,  reliability  spec¬ 
ifications.  For  many  items  in  our  30-day  combat  scenario, 
the  effects  of  longevity  on  item  loss  may  be  negligible, 
and  the  constant  loss  rate  structure  as  we  have  employed 
it  not  a  source  of  excessive  concern. 

In  the  next  section  we  will  look  in  more  detail  at 
a  procedure  by  which  professional  judgement  could  be  used 
collectively  to  estimate  or  verify  CARFs  or  MTTLs. 


Ill,  CARF  GENERATION  AND  VERIFICATION 
USING  MILITARY  JUDGEMENT 

A  frequent  procedure  for  eliciting  expert  opinion 
from  knowledgeable  individuals  is  that  of  asking  them  to 
do  some  form  of  ordinal  rating  of  various  items  with  regard 
to  some  property.  Judges  usually  can  provide  ordinal  ratings 
or  ranking  with  much  greater  confidence  than  they  can  state 
numerical  values.  For  example,  from  professional  expertise 
one  should  be  able  to  state  which  of  two  items  should  have 
the  greater  CARF  value,  and  do  this  with  more  assurance  than 
to  state  estimated  CARF  values  for  each  of  the  two  items. 

The  method  given  on  the  following  pages  allows  ordinal 

information  furnished  by  judges  to  be  combined  with  a  model 

of  judge  behavior  to  obtain  an  interval  scale.  Thus  while 

judges  may  be  asked  only  to  rank  items  in  terms,  say,  of 

their  loss  rates,  the  collective  inputs  of  many  judges  permit 

and  interval  scale  to  be  inferred  without  arbitrary  scoring.^ 
Then,  specification  of  CARF  values  for  two  of  the  items 

should  yield  the  needed  ratio-scaled  CARF  estimates  for 

all  the  items  being  considered. 

Although  ordinal  ratings  by  judges  may  arise  in  many 
diverse  situations,  in  too  may  cases  such  data  is  gathered 


without  a  clear  notion  of  how  it  will  be  processed  in  order 
to  have  a  resulting  scale  that  collectively  represents 
the  inputs  of  the  judges.  , 

A  commonly  used  (but  not  recommended)  approach  to 
dealing  with  a  set  of  lists  of  rankings  is  to  award  arbit¬ 
rarily  chosen  numerical  values  for  various  ranking  positions, 
and  then  to  compute  an  average  score  for  each  item.  For 
example,  if  there  are  ten  items  to  be  scaled,  one  might 
award  nine  points  if  a  judge  ranks  an  item  first,  eight 
points  if  he  ranks  it  second,  and  so  on.  The  procedure 
imposes  an  interval  scale  interpretation  upon  the  ordinal 
data  furnished  by  the  judges,  and  in  most  cases  is  very 
difficult  to  justify.  Further,  from  a  sensitivity  analysis 
point  of  view,  it  may  be  shown  that  the  results,  and  even 
their  rank  order,  often  depend  dramatically  upon  rhe 
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scoring  system  used. 

On  the  following  pages  we  will  first  talk  about 
various  ways  to  obtain  expert  judgements  from  knowledgeable 
persons.  Then,  we  will  present  a  step-by-step  procedure 
to  convert  the  rankings  to  an  interval,  and  then  to  a  ratio 
scale  permitting  CARF  estimation.  The  mathematics  supporting 
the  procedure  are  presented  in  Appendix  B . 


Obtaining  InforTnation  From  Judges 

There  are  several  dif:^erent  ways  of  obtaining  rankings 
from  judges,  and  when  selecting  a  particular  approach  one 
would  want  to  consider  both  the  nu*.i)er  of  items  for  which 
values  are  sought,  and  the  number  of  judges  who  are  to  be 
queried.  Tradeoffs  may  have  to  be  made  between  the  effort 
that  will  be  required  of  a  judge  and  the  amount  of  confid¬ 
ence  one  wishes  to  have  in  the  resulting  scale  values. 

We  begin  by  discussing  various  ways  in  which  judges 
might  record  a  full  ranking  of  the  items.  Then,  we  will 
describe  a  paired  comparisons  approach  used  by  psychologists 
and  others. 


Ranking  all  the  Items 

A  direct  scheme  for  querying  a  judge  is  to  present 
him  with  a  set  of  items  together  with  instructions  that 
he  is  to  rank  them  in  terms  of  their  CARF  values.  (Alter¬ 
natively,  we  could  ask  him  to  do  thi  ranking  in  terms  of 
MTTL,  or  loss  rate,  or  chance  of  loss.)  As  part  of  these 
instructions,  he  should  be  encouraged  to  rearrange  and 
adjust  his  ordering  until  he  is  satisfied  with  it. 


One  way  for  the  judge  to  exp  ess  his  rankings  is  to 
write  the  names  of  the  items  in  a  ranked  list.  The  list 
gives  him  and  others  an  easily  inspected  display  of  his 


work,  and  thereby  facilitates  any  subsequent  rearrangement 
he  may  wish  to  do.  Rather  than  writing  the  names  of  the 
items,  the  judge  could  write  symbols (numbers  or  letters) 
representing  the  items.  (For  example,  he  would  place  item 
Number  4  first,  item  Number  1  second,  and  so  on.)  This  is 
easy  to  do,  but  his  opportunity  for  easy  inspection  is 
hindered  by  the  coding.  Further,  unless  instructions 
are  very  clear,  there  can  be  ambiguity  about  the  results 
in  that  if  a  3  is  listed  first,  ho  might  have  meant  that 
item  Number  1  was  third,  rather  than  item  Number  3  first. 

It  is  generally  felt  better  to  ask  the  judge  to  use  letter 
symbols  or  abbreviations  of  the  item  names.  Also,  it  is 
essential  to  make  clear  which  goes  at  the  top  of  the  list; 
the  item  with  the  highest  chance  of  loss,  or  the  item  with 
the  lowest  chance  of  loss. 

A  different  procedure  asks  the  judge  to  work  directly 
with  the  given  list  of  items,  indicating  on  that  list  which 
is  ranked  first,  second,  and  so  on.  This  approach  makes  it 
very  difficult  for  the  judge  to  inspect  his  work  to  see  if 
he  is  satisfied  with  the  rankings  he  has  given. 

A  very  efficient  scheme  tending  to  encourage  rearrange¬ 
ment  and  to  eliminate  logic  errors  is  to  have  the  items 
written  on  cards  -  -  judges  spread  out  the  cards  and  arrange 
them  to  their  satisfaction. 

Occasionally,  a  judge  will  feel  that  he  cannot  discrim- 


inate  between  two  items  in  terms  of  their  chance  of  loss. 


Also,  there  will  be  cases  where  a  judge  wishes  not  to  rank 
an  item  because  he  feels  he  does  not  know  enough  about  it. 
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Although  one  will  wish  to  encourage  judges  to  rank  all 
items  to  the  extent  that  they  are  able,  it  is  undesirable 
to  require  them  to  furnish  information  which  they  do  not 
feel  they  can  express.  In  the  method  given  in  this  report, 
information  from  judges  who  omit  items  or  who  list  ties 
may  be  used  completely. 

The  Method  of  Paired  Comparisons 

A  method  of  obtaining  ordinal  information  from  judges 
which  has  received  considerable  attention  in  the  literature 
of  Psychology  is  the  Method  of  Paired  Comparisons ,  >  16 

Here  a  judge  is  presented  with  pairs  of  items  and  is  asked 
to  indicate  (mark,  circle,  etc.)  which  item  in  the  pair 
has  the  higher  loss  rate.  This  data  may  be  incorporated 
directly  into  the  scaling  procedure  that  will  be  presented 
here.  If  there  are  n  items  to  be  rated,  this  involves 
n{n-l)/2  comparisons,  so  that  if  there  are  20  items,  the 
judge  would  look  at  190  pairs.  Aggregate  rankings,  which 
are  inherent  in  the  total  ranking  approach,  will  not  be 
possible.  Ties  are  treated  as  omitted  comparisons. 

Apart  from  the  large  number  of  comparisons  that 
may  have  to  be  made,  another  problem  with  paired  comparisons 
is  that  of  insuring  logical  consistency.  A  judge  might 
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rank  A  over  B,  B  over  C,  and  then  C  over  A.  It  may  be 
argued  that  such  an  inconsistency  is  properly  part  of  the 
judge's  expression  of  his  feelings,  and  should  be  accepted. 
The  practical  problem  that  remains  is  the  judge's  efforts 
to  try  to  avoid  this,  even  if  asked  not  to.  This  adds  to 
the  busywork  of  his  task,  and  may  influence  the  spontaneity 
of  his  ratings. 


Procedure  for  Obtaining'  Scale  Values 

1.  Data  from  the  judges'  ratings  (ordinal)  may  be 
tallied  in  an  f^j  array,  where  f^^  is  the  number  of  judges 


who  ranked  item  j  as  possessing  a  greater,  say,  loss  rate 


than  item  i.  When  this  is  done,  f  .  +  fu  will  be  the 

ab  ba 


number  of  judges  who  compared  item  a  with  item  b. 


2.  The  p^j  array  may  be  computed  from  the  f^^  array 


as  the  proportion  of  judges  comparing  i  and  j  who  rated 
item  j  as  having  a  greater  loss  rate  than  item  i.  Here, 


p.  .  =  ^^3 

^3  f..  +  f.. 


(11 


11 


11 


Note  that  p^j^  +  p^^  =  1.  On  the  diagonal  of  the  p^^  array. 


ba 
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we  set  0.5 


3.  Using  a  table  of  the  standard  normal  distribution 


F(z),  we  prepare  an  array  where  z^^  is  the  standard 


normal  value  conresponding  to  Pj^j#  or 


V.  =  j  _f(z)d.  =  F(Z:  .)  . 


We  omit  (leave  blank)  entries  corresponding  to  0.98, 

or  p. .<  0.02.  The  z. .  array  will  have  zeros  on  the  main 
ij  ID 

diagonal,  and  if  a  and  b  are  items,  z^j^  =  - 

4.  If  the  z^j  array  has  no  missing  values,  we  use  the 
column  averages  as  the  preliminary  scale  values  for  the 
n  items.  These  column  averages 


1  ( 

I  7. .  . 

S  .  -  i  =  V  .  j  -  l ,  2  ,  .  .  .  ,  n  . 


are  the  loss  rate  values  for  the  n  items  on  an  interval 
scale, that  is,  a  scale  with  arbitrary  origin  and  unit. 

5.  If  the  z^j  array  has  missing  values,  the  procedure 
is  as  follows.  We  compute  column  averages  for  those 
columns  which  are  complete,  and  use  these  column  averages 
as  scale  values  for  those  items.  For  incomplete  columns, 
we  write  a  set  of  linear  equations  of  the  form 


n  .  S  .  -  T  S  .  =  y  ?: ;  ■ 

i  ^  h  .  1  *  I  . 

J  J 


where  1)^  denotes  the  sot  of  n^  elements  in  column  j  of  the 

array.  Substituting"  the  scale  values  already  determined 
by  averaging  values  in  the  complete  columns,  we  solve  the 
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set  of  simultaneous  equations  to  obtain  the  remaining 
scale  values. 

6.  The  preliminary  scale  values  which  result  from 
the  foregoing  are  on  an  interval  scale,  and  may  be  changed 
to  other  interval  scales  by  changing  origin  and  unit 
via  any  linear  transformation  of  the  form 

s!  =  A  +  BSj,  B>0  .  (14) 

To  obtain  the  final  scale  values  we  need  to  add  information 
which  will  specify  the  proper  unit  and  origin.  This  can 
be  done  by  estimating  final  scale  values  for  exactly  two 
of  the  items,  call  them  g  and  h.  (For  example,  we  might 
say  that  the  loss  rate  for  item  g  is  L^=  10,  and  the  loss 
rate  for  item  h  is  L.®  3.)  Then,  the  final  scale  L.  is 
obtained  by  transforming  all  the  preliminary  scale  values 


(15) 


The  values  found  by  (15)  will  be  the  end  result  of  the  judges 
ranking  of  the  items.  If  items  were  ranked  in  terms  of 
their  CARF  values,  then  after  estimating  CARF  values  for 
two  items  g  and  h.  Equation  (15)  yields  the  estimated 
CARF  values  for  all  of  the  items.  One  useful  way  to 
employ  this  method  is  to  add  to  the  list  of  items  which 
are  to  be  scaled  to  obtain  CARF  values  two  additional 


items  for  which  CARF  values  exist  and  are  felt  to  be 


satisfactory.  Judges  rank  all  the  items,  and  at  the  end 
of  the  data  analysis  the  two  items  with  knownCARFs  are 
used  to  position  the  scale. 

Remarks 

The  procedure  presented  here  employs  collective 
military  judgement  to  generate  estimates  of  factors 
describing  loss  properties  of  various  types  of  equipment 
Input  data  is  simply  in  the  form  of  ranked  lists,  while 
the  final  values  are  essentially  ratio  scaled  when  the 
procedure  is  used  to  estimate  such  factors  as  loss  rates 
MTTL  values,  or  CARFs. 

The  method  has  the  disadvantage  that  it  is  based 
upon  disagreement  among  judges;  if  judges  are  in  close 
agreement  in  their  ranked  lists,  many  items  will  be 
unsealed  except  in  an  ordinal  sense.  A  related  concern 
is  that  a  reasonably  large  number  of  judges  is  needed. 
One  cannot,  for  example,  expect  much  success  if  only 
nine  or  ten  judges  are  available. 

A  demonstration  of  this  procedure  employing  real 
judges  and  data  is  given  in  the  next  section,  where  we 
look  at  obtaining  values  for  chance  of  loss  for  21 
functional  areas  of  equipment. 


IV.  DEMONSTRATION  OF  THE  METHOD: 


CHANCE  OF  LOSS  FOR  TWENTY-ONE 
FUNCTIONAL  AREAS 


As  a  demonstration  of  the  use  of  military  judgement  to 
estimate  chance  of  loss  of  various  kinds  of  equipment, 
twenty-three  knowledgeable  judges  were  asked  to  rank 
twenty-one  types  of  equipment  in  terms  of  "chance  of  loss 
of  individual  items".  The  equipment  types  were  identified 
by  management  function  area  codes,  and  the  list  of  items  is 
given  in  Table  2. 

Raw  Data 

A  replica  of  the  ranking  form  is  shown  in  Table  3. 

Judges  wrote  area  code  numbers  on  their  lists,  rather  than 
equipment  names .  Of  the  twenty-three  judges,  only  five  rated 
all  the  items;  most  omitted  two  or  three,  although  one 
ranked  only  five  items. 

The  instructions  said  nothing  about  writing  ties  on  the 
same  line,  but  seven  of  the  judges  did  this.  Since  the 
scaling  procedure  handles  ties  and  omitted  items,  all  the 
data  furnished  by  the  judges  was  used. 

It  is  a  characteristic  of  the  scaling  method  that  if 
all  judges  rank  an  item  first  (or  all  rank  it  last) ,  then 
that  item  cannot  be  scaled  except  in  an  ordinal  sense. 
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TABLE  2.  Table  of  Management 
Functional  Area  Codes 


FA  Code 


Description 


10 

13 

14 
16 
17 

19 

20 
21 
23 
26 

29 

30 
35 

40 

41 

42 

43 
46 

49 

50 
11 


Radios 

Air  Command/Control  Equipment 
Air  Support  Radar/IFF  Equipment 
Electronic  Equipment 
Ground  Support  Radar 
Intelligence/Surveillance  Equipment 
Generators 

Environmental  Control  Equipment 
Earth  Moving  Equipment 
Materials  Handling  Equipment 
Engineer  Support 
Trucks 

Towed  Motor  Transport  Equipment 
Tanks 

Amphibious  Assault  Vehicles 

Light  Armored  Vehicles 

Artillery 

Infantry  Weapons 

Missile  Systems 

High  Density/Low  Deadline 

Communications  Support  Equipment 


TABLE  3.  Demonstration  Questionaire 

« 

Please  rank  the  listed  functional  HIGHEST  CHANCE 

OF  LOSS 

areas  in  terms  of  the  chance  of 
loss  of  individual  items  in  the 
functional  area.  For  example, 
if  you  think  that  Towed  Motor 
Transport  Equipment  has  the  high¬ 
est  chance  of  loss,  you  would 
write  the  name,  or  the  code  '35' 
on  the  top  line.  Omit  any 
functional  areas  you  are  unable 
to  rank , 


LOWEST  CHANCE  OF  LOSS 


I 
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This  was  the  case  with  Item  50,  High  Density/Low  Density. 

It  was  ranked  by  fewer  judges  (nine)  than  any  other  item, 
and  most  placed  it  last  on,  their  lists.  No  two  judges  ranked 
it  above  any  other  specific  item.  As  a  result,  it  was 
dropped  from  the  analysis,  to  reappear  in  the  final  scale 
where  it  would  be  declared  as  having  a  lower  chance  of 
loss  than  the  lowest  item  scaled. 

Of  the  remaining  twenty  items,  judge's  omissions 
were  scattered  over  the  items.  The  exception  was  Item  11, 
Communication  Support  Equipment,  which  was  ranked  by  only 
ten  of  the  twenty-three  judges.  Since  frequencies  were  good 
on  the  other  items.  Item  11  was  allowed  to  remain  in  the 
analysis. 

Obtaining  the  Interval  Scale 

Scaling  twenty  items  simultaneously  is  a  larger  than 
typical  problem,  and  it  was  necessary  to  extract  paired 
information  from  the  lists  for  the  f^^  array  by  using  a 
computer,  since  the  task  was  too  large  to  be  done  by 
manual  methods.  When  cells  with  p^^  values  which  were  too 
large  or  too  small  were  deleted,  there  remained  six  com¬ 
plete  columns  out  of  the  original  twenty.  This  meant  that 
six  items  could  have  their  scale  values  computed  easily,  and 
then  a  set  of  14  simultaneous  linear  equations  would  have 
to  be  solved  to  obtain  the  remaining  scale  values. 


An  easier  approach  was  taken.  The  data  was  divided 
into  two  scaling  problems,  each  with  eleven  items  so  that 
there  were  two  in  common.  The  sets  of  items  were: 

#1:  10,13,14,16,17,19,20,40,41,42,43;  and 
#2:  10,11,16,21,23,26,29,30,35,46,49. 

Scales  for  the  items  in  each  problem  were  found  using  the 
procedures  described  in  Section  III  of  this  report,  and 
then  the  two  scales  were  merged  by  a  linear  transformation 
based  upon  the  two  common  items. 

This  resulted  in  an  interval  scale  for  chance  of  loss 
for  the  20  equipment  items;  the  scale  is  shown  in  Figure  1 
Note  that  no  numbers  appear  on  the  scale  since  they  would 
be  misleading:  scale  unit  and  origin  are,  at  this  point, 
totally  arbitrary. 

Reference  Points 

Two  of  the  items  were  assigned  chance  of  loss  values: 
Trucks  at  0.0480,  and  Tanks  at  0.3902.  These  values  were 
specified  independently  of  the  ranking  information. 
Interval  scale  values  for  the  entire  set  of  equipment 
areas  as  shown  in  Table  4  were  transformed  to  a  scale 
that  would  include  the  two  reference  points.  This  was 
done  via  Equation  (15),  which  in  this  case  was 


L.  =  -0.4658  +  O.OIS  . 
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TABLE  4.  Scaling  Results  for 
Twenty-one  Equipment  Functional 
Areas,  Chance  of  Loss 


Chance  of  Loss, 
based  upon: 


FA  Code 

Interval  Scale 
High  90,  Low  10 

L3Q  =  0.0480 
L^q  =  0.3902 

10 

48.9 

0.0221 

11  . 

41.1 

-0.0558 

13 

82.6 

0.3583 

14 

67.1 

0.2036 

16 

44.1 

-0.0258 

17 

44.5 

-0.0218 

19 

34.6 

-0.1206 

20 

10.0 

-0.3660 

21 

21.9 

-0.2473 

23 

33.9 

-0.1276 

26 

29.3 

-0.1735 

29 

37.3 

-0.0936 

30 

51.2 

0.0480* 

35 

37.2 

-0.0947 

40 

85.8 

0.3902* 

41 

90.0 

0.4321 

42 

75.5 

0.2874 

43 

63.7 

0.1697 

46 

36.9 

-0.0976 

49 

47.1 

0.0041 

50 

.  Below  10 

Below  -0. 3 

The  resulting  values  are  shown  in  Table  4. 

The  presence  of  negative  values  in  the  right-hand 
column  of  Table  4  deserves  comment,  since  the  terms  "chance 
of  loss"  could  be  thought  of  a  denoting  a  probability,  and 
thus  a  non-negative  value.  In  fact,  there  is  nothing  in 
this  method  to  restrict  values  from  being  negative.  Had 
the  scale  results  all  been  positive,  they  probably  would 
have  been  interpreted  as  probabilities  without  thought 
of  justification.  The  presence  of  negative  values  serves 
as  a  useful  reminder  that  "chance  of  loss"  need  not 
necessarily  be  a  probability. 

On  the  other  hand,  if  one  insists  that  probability 
values  should  result,  then  one  can  ask  where  errors  might 
lie.  The  basic  question  when  attempting  to  confirm  a 
value  by  generating  it  from  a  different  source  is  what  to  do 
when  the  two  values  differ.  (Agreement  supports  the  existing 
value.)  In  this  case,  if  we  insist  that  the  chance  of  loss 
values  should  lie  between  zero  and  1.0,  then  judge  and 
reference  values  are  in  disagreement:  one,  the  other,  or 
both  are  in  error  to  a  conspicuous  extent. 

We  note  that  the  judges  rated  14  of  the  21  items  (more 
than  half)  as  having  a  chance  of  loss  lower  than  Trucks, 
and  that  the  reference  value  for  truck,  0.048,  was,  as  a 
probability,  already  a  low  value.  Had  the  reference  value 
for  Trucks  been  higher,  positive  scale  values  would  have 


resulted. 


One  way  to  investigate  this  would  be  to  repeat  the 
study,  possibly  vrlth  a  fewer  number  of  items,  but  retaining 
#30,  Trucks,  and  #40,  Tanks.  Then, to  the  list  of  items  that 
judges  would  be  asked  to  rank  we  would  add  two,  described 
as  "A  functional  area  whose  chance  of  loss  is  0.048"  and 
"A  functional  area  whose  chance  of  loss  is  0.39".  If  these 
values  are  consistent  with  military  judgement,  about  half 
the  judges  would  rank  trucks  above  0.048,  and  about  half 
below.  Similarly  for  tanks,  about  half  would  rank  tanks 
above  0.39,  and  about  half  below. 

Finally 

This  demonstration  served  as  a  first  effort  to  use 
collective  rankings  to  value  factors  such  as  chance  of 
loss.  The  data  provided  a  judgement-based  interval 
scale  for  twenty  types  of  equipment,  and  disagreed  in  a 
ratio  scale  sense  with  two  reference  points. 

It  is  hoped  that  the  work  presented  here  will  be 


useful  to  those  interested  in  applying  judgemental  methods 
to  the  verification  of  CARFs  and  similar  measures. 


APPENDIX  A 


SOME  CAUTIONS  REGARDING  CARF  GENERATION  AND  USE 

There  are  many  ways  to  estimate  a  CARF  value,  but  in 
essence  the  estimate  of  losses  upon  which  the  CARF  is  based 
will  be  in  one  of  two  basic  forms.  Either  the  entire  loss 
ratio  will  be  estimated  (as  a  percentage,  proportion,  or 
a  fraction) ,  or  the  number  of  items  that  will  be  lost  will 
be  estimated. 

In  the  case  where  the  loss  ratio  is  estimated,  the 
denominator  of  that  ratio  is  left  implied.  An  example  would 
be  an  estimate  using  military  judgement,  "I  think  we'd 
lose  10%  of  that  kind  of  equipment".  Here  it  is  not  clear 
what  quantity  the  10%  refers  to.  It  is  tempting  to  assume 
that  the  reference  is  to  the  T/E  allowance  for  forces  in 
contact  with  the  enemy,  consistent  with  CARF  definition 
and  interpretion.  A  person  providing  the  estimate  who 
has  worked  with  values  reflecting  the  quantity  of  item  in 
the  FMF,  however,  might  be  thinking  of  that  reference  when 
he  states  10%,  while  a  company  commander  might  be  using  a 
different  reference.  Generally,  when  estimates  are  given 
directly  as  loss  ratios,  there  is  no  guarantee  that  the 
reference  will  be  the  desired  one  unless  this  is  made  very 


clear  to  the  individual  making  the  estimate.  Errors  can 
be  substantial.  Also,  this  kind  of  estimate  probably 

I 

requires  knowledge  of  the  in-use  amount. 

An  alternative  form  of  loss  estimate  upon  which  a 
CARF  might  be  based  is  the  number  of  items  lost  and  requiring 
replacement  in  a  30-day  period.  This  is  the  form  of  estimate 
that  would  result  when  a  computerized  combat  model  is  used 
to  produce  a  CARF  value,  A  direct  statement  of  losses  could 
also  come  from  military  judgement,  e.g.,  ”I  think  we'd  lose 
two" .  Here  the  stated  estimate  is  independent  of  the 
reference  value,  at  least  explicitly,  and  thus  the  denom¬ 
inator  for  the  CARF  is  left  to  be  specified  separately. 

Figure  2  shows  the  anatomy  of  a  CARF,  detailing  the 
components  involved  in  producing  a  number  which  satisfies 
the  definition  of  a  replacement  factor, viz.,  the  expected 
percentage  of  equipment  that  will  require  replacement 
.  .  .  for  a  30-day  period.  Our  interest  here  is  not  only 
the  numerator  and  denominator,  but  also  the  fact  that  the 
definition  requires  a  CARF  to  be  a  percentage.  Real  and 
potential  difficulties  with  both  of  these  characteristics 
of  a  CARF  are  identified  and  discussed  in  this  appendix. 


The  Denominator:  The  "In-Use"  Amount 


A  CARF  is  a  percentage,  and  as  such  is  essentially 
dimensionless.  When  used  for  planning  and  other  purposes. 


The  CARF  estimation  problem 
lies  primarily  in  obtaining 
realistic  values  for  losses. 


CARF  = 


m  •« 

Losses;  the  number  of  items 

that  will  require  replacement 
during  a  30-day  period 

100 

Number  of  items  in  use 

^  m 

m  • 

Failure  to  place  correct  values  here 
can  lead  to  CARFs  which  are  substan¬ 
tially  in  error,  even  though  losses 
are  well  estimated. 


FIGURE  2. 


The  Anatomy  of  a  CARF 
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its  most  frequent  application  will  be  that  of  multiplying 
a  quantity  of  equipment  to  establish  the  quantity  that  will 
need  replacement.  Accordingly,  it  is  crucial  if  errors 
are  to  be  avoided  that  the  quantity  the  CARF  multiplies 
be  of  the  same  substance  as  the  denominator  used  to  detemine 
the  CARF  value. 

There  are  several  ways  to  interpret  in-use  amount, 

and  as  suggested  above,  clarification  might  usefully  be 

based  upon  CARF  application.  This  appears  to  have  been  done 

2 

as  follows,  quoting  from  the  War  Reserve  Policy  Manual . 

1.  "The  combat  active  replacement  factor  will  be 
applied  for  units  during  those  periods  when  they 
are  actually  in  active  combat  operations.  A  force 
in  contact  with  the  enemy  is  considered  to  be  in 
active  combat."  {pg  4-3, 4-4) 

2.  "When  applied  to  the  density  of  equipment,  the 
combat  active  replacement  factor  determines  the 
amount  of  equipment  which  must  be  replaced  each 
month  to  maintain  the  full  T/E  allowance. " (pg  4-4) 

3.  "  .  .  .  (CARFs)  multiplied  by  the  equipment 
density  or  unit  Table  of  Equipment  (T/E) 
allowance."  (pg  12-5) 


As  has  been  said,  as  long  as  the  denominator  used  in 
specifying  the  CARF  and  the  quantity  the  CARF  multiplies 
are  of  the  same  nature,  there  will  be  no  difficulty. 
However,  we  will  have  errors  that  may  be  substantial  when 


a  CARF  is  used  to  multiply,  say,  the  number  of  items  in 
the  FMF. 

Another  area  where  there  may  be  difficulty  with  the 

i 

CARF  denominator  occurs  when  a  CARF  is  based  upon  an  Army 

replacement  factor,  called  a  WARF.  The  Army's 

"Wartime  Replacement  Factor  (WARF)  is  the 
average  daily  catastrophic  (nonrepairable) 
item  loss  rate  expressed  as  a  percent  of 
the  average  authorized  item  strength  in 
the  combat  theater". 

To  let  CARF=(WARF)  (30)  means  (assxaming  all  other  aspects 
of  Army  and  Marine  operations  to  be  similar)  that  "average 
authorized  item  strength  in  the  combat  theater"  is  the  same 
concept  as  the  Marine's  "in-use  amount". 

Percentage  or  Proportion? 

Current  numerical  values  for  CARFs  for  various  items 

3 

are  listed  in  the  Table  of  Authorized  Materiel  (TAM) . 
Consistent  with  definitions  given  elsewhere,  the  TAM  defines 

replacement  factors  as  percentages  and  then  lists  the 

values  for  "replacement  factors  for  30  days"  beginning  on 

page  26-1.  However,  the  values  that  are  listed  are  in  fact 

proportions,  100  times  smaller  than  the  replacement  factors 

described.  It  is  apparently  true  that  Supply  Officers  have 

been  trained  to  interpret  these  values  as  proportions. 

However,  others  seeking  CARF  values  will  probably  go  to  the 

TAM  as  the  basic  and  most  available  source.  The  TAM  says 

clearly  that  they  are  percentages,  and  thus  for  unknowing 


APPENDIX  B 


DEVELOPMENT  OF  THE  RANKING  METHOD 


The  procedures  in  Section  II  of  this  report  employed  a  ranking 
method  by  which  experts  (called  judges)  expressed  their  feelings  about 
loss  rates  or  CARF  values  (scale  values  or  amount  of  the  parameter) 
associated  with  various  items  of  equipment  (instances))  by  ranking 
the  items.  The  collective  rankings  of  the  judges  were  then  used 
to  produce  interval  scaled  values  for  the  amount  of  the  property 
possessed  by  each  of  the  instances. 

There  are  several  ways  to  approach  scale  development  from  ordinal 
data.  Models  vary,  depending  upon  what  assumptions  are  made.  We  shall 
present  what  is  probably  the  most  widely  used  approach.  We  assume 

the  following: 

a.  A  judge  cannot  directly  express  his  feelings  about  the 
scale  value  of  instance  j,  but  is  able  to  rank  instances  in 
accordance  with  his  feelings. 


b. 


Over  the  population  of  judges, 
random  variable. 


is  a  normally  distributed 


c.  All  instances  possess  the  same  variance*  for  x,  so  that 


0 


2 

i 
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4 


I 
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d.  The  correlation  coefficient  for  x  between  any  pair  of 

instances  is  the  same,  so  that  p  =  p . 

From  these  assumptions  we  may  deduce  the  following.  Let  i  and  j  be  two 

instances.  A  judge's  feeling  about  the  amount  of  the  property  possessed 

by  instance  i  is  a  normally  distributed  random  variable  with  mean 

2 

and  variance  o  ,  and  a  judge's  feelings  about  the  amount  of  the  proper! 

possessed  by  instance  j  is  normally  distributed  random  variable  x.  with 

2 

mean  Sj  and  variance  o  .  Since  the  difference  between  two  normally 

distributed  random  variables  is  itself  a  normally  distributed  random 

2  2  2 

variable,  (Xj^-Xj)  is  normal  with  mean  S^-S^  and  variance  o  +  o  -2po  = 

2 

2a  (1-p)  where  p  is  the  correlation  coefficient.  Now,  the  probability 
that  a  judge  rates  instance  j  as  possessing  more  of  the  property  than 

instance  i  may  be  expressed  as  Pr(Xj  >  x^) .  We  will  operate  on  the 
inequality  as  follows; 


Pr(Xj  >  x^)  =  Pr  (0  >  x^  -  x^) 


=  Pr 


-(S.  -  S.)  >  (X.  -  X.)  -  (S.  -  S.) 

1  J  1  J  1  ] 


S.  S.  (x.  -  X.)  -  (S.  -  S.) 

=  Pr  J--  - i - L. 

^2a^  (1-p)  /2o^(l  -  p) 


The  right  hand  side  of  the  final  inequality  above  is  a  normally  distri¬ 
buted  random  variable  with  a  mean  of  zero  and  a  variance  of  unity.  Thus 


we  have 


L6 


where  z  is  the  standard  normal  deviate. 

An  estimate  of  the  left-hand  side  of  (16)  may  be  obtained  from  the 
ranking  information  furnished  by  the  judges.  The  proportion  of  judges 
who  rank  instance  j  as  possessing  more  of  the  property  than  instance  i 
may  be  used  as  an  estimate  of  Pr(Xj  >  x^) . 

Let  p^j  be  the  proportion  of  judges  who  rate  instance  j  as  possess 

ing  more  of  the  property  to  be  scaled  than  instance  i.  Let  z^^  be  the 

value  of  the  standard  normal  deviate  (from  the  Normal  Table)  associated 

with  p. that  is,  z. .  is  the  value  of  z  for  which  the  leftward  area 
^  J  ^  J 

under  the  normal  N(z;0,l)  curve  is  p. . 

•‘•J  • 

Examples  are 


Pij 

^ij 

0.92 

1.645 

0.90 

1.282 

0.50 

0 

0.001 

-3.0 

We  now  have,  from  Equation  (16)  estimating  equations  of  the  form 


S.  -  S. 

^li  -  ,JL_.  1  _ 

-  p)  • 


with  one  of  these  equations  occurring  for  each  instance  pair  i,j.  In 
(17), the  lefc  hand  z.^  values  come  from  the  judges'  rankings,  being  the 
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standard  normal  daviate  associated  with  the  proportion  of  judges  who 
ranked  instance  j  as  possessing  more  of  the  property  than  instance  i. 

On  the  right  hand  side  of  (17)  » we* have  Sj~S^,  the  difference  in  two 
of  the  scale  values  we  wish  to  obtain. 

Since  we  seek  values  of  and  oh  an  interval  scale  whose  unit 
and  origin  are  yet  unspecified,  we  may  use  our  freedom  to  specify  unit 
and  origin  to  simplify  the  mathematical  development.  Reserving  speci¬ 
fication  of  the  scale's  origin  until  later,  we  can  currently  obtain  a 
simpler  form  of  (17)  by  specifying  a  unit  for  the  scale  such  that 


'^2o^(l-p)  =  1. 


This  is,  of  course,  not  an  assumption. 

The  scaling  problem  now  stands  as  follows.  We  have  n  instances 

to  be  scaled;  we  seek  values  for  S  ,  S., .  . . . ,  S  .  We  have  an  n  x  n 

12  n 

array  of  z^j  values  which  came  from  judges’  rankings,  and  we  have  a 
set  of  n(n-l)/2  estimates  of  the  form 


z .  .  S  .-S . 

1]  “V-  ]  1 


(19) 


For  n  >  3,  we  will  have  more  estimates  than  values  to  be  estimated 

Viewing  the  various  estimates  as  being  equally  useful,  we  will  now 

propose  a  least-squares  fit  over  the  estimates  {'19)  . 

Before  going  ahead  with  the  least-squares  procedure,  we  must  at 

this  time  pause  to  point  out  an  important  characteristic  of  the  z^^  array 

If  all  judges  rank,  say,  instance  a  as  posessing  more  of  the  property 

than  instance  b,  then  we  will  have  p^^  =  1.0,  p^^^  =  0,  and  thus  z^^  =  « 

and  z  .  =  -  In  practice,  to  avoid  numerical  bias  by  a  small  number 
ao 


of  judges,  .  values  corresponding  to 


p.  .  >  0.98 


p .  .  <  0.02 


are  omitted  from  the  array 


Thus  (if  any)  there  will  be  an  even 


number  of  "holes”  in  the  z^^  array,  symmetric  about  the  diagonal.  Our 
continued  development  must  take  into  account  the  possibility  that  the 
Zj^j  array  may  be  incomplete. 

Least-Squares  Solution 

For  the  least-squares  procedure,  let  ij)j  be  the  set  of  rows  possess 
ing  elements  in  the  jth  column  of  the  z^j  array.  Now  ideally,  z^j  = 

S.  -S.,  and  thus  for  our  least-squares  fit,  we  will  want  to  find  a  value 
for  Sj  that  minimizes 


W.  = 
3 


y  (z. .  -  (S .  -  s. ^  . 

ui.  \  ^  ^  ’ 


The  algebraic  expansion  of  the  summation  term  is 


W.  = 
3 


I  [z,.  -  .  2z,.S,  .  S.2  -  2S.S.  * 


I  z..  -2S.  I  z..+2y  Z..S.  +  n.S.-2S.  'j  S.  +  Ts 

0^-1  -It#  1-1  L  iTi  'll  -i  L  t 
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Equation  (20)  represents  a  set  of  n  linear  equations  in  S,,  S»,  S  . 

12  n 

It  may  be  shown  that  this  set  of  equations  is  not  independent  (the  reader 

may  wish  to  establish  this  by  showing  that  the  negative  of  any  equation 

is  equal  to  the  sum  of  all  remaining  equations  ,  or  that  the  coefficient 

matrix  is  singular) .  The  set  of  equations  becomes  independent  when  any 

one  equation  is  eliminated,  leaving  us  with  a  set  of  n-1  equations  in  n 

unknowns.  After  arbitrarily  setting  a  value  for  one  of  the  (which  is 

equivalent  to  setting  the  origin  for  the  interval  scale  we  are  developing) , 

we  can  solve  the  set  of  equations  using  standard  procedures  such  as 

17  . 

Gauss-Jordan  elimination  methods. 

Solving  sets  of  simultaneous  equations  becomes  a  cumbersome 
activity  when  appropriate  computing  equipment  is  unavailable  and  when 
n  is  even  modestly  large.  This  is  true  even  when  the  coefficient  matrix 
consists  of  small  integers,  as  is  our  case.  A  method  is  given  below  to 
reduce  this  effort  by  decreasing  the  number  of  equations  which  have  to 
be  fully  solved.  Savings  in  effort  will  occur  only  when  there  are  columns 


ill  th6  j  srray  which  are  complete  in  that  they  have  no  missing  elements  • 
To  present  our  labor-saving  approach,  we  return  to  our  set  of  n 
equations  of  the  form  • 


n  .S  . 
D  D 


-  .1  ^  = 


I  2^.  ;  .  1,2, 


. ,  n, 


(20)  ^ 


If  there  exists  one  or  more  complete  columns  in  the  z. .  array,  we  precede 
by  establishing  the  origin  for  the  scale  by  making  the  average  scale  value 


zero,  1. e. , 


1=1  -  0 


which  implies  that  I  S  =0 

iil  " 


Suppose  that  column  k  is  complete,  so  that  n.  =n.  The  equation  for  column 

n  n  ^ 

k  is  "  ^  S.  -  I  _ 

i=l  i=l 


and  immediately. 


^k  = 


n 

J  .^ik 
1-1 


since  X  S-  =  0  .  Thus,  when  a  column  is  complete,  we  can  immediately 

i=l 

compute  the  s^ale  value  for  the  instance  associated  with  that  column  as 
the  average  of  the  z^^  in  that  column.  After  we  have  done  this  for  all 
complete  columns,  there  will  remain  a  set  of  incomplete  columns  associated 


with  instances  for  which  we  do  not  yet  have  the  scale  values.  Ml 
that  remains  is  to  (a)  write  out  the  equations  of  the  form  of  (20)  for 
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the  incomplete  columns/  substitute  the  values  of  the  s .  we  have  obtained 
from  the  complete  columns/  and  (b)  then  solve  this  smaller,  easier  set 
of  equations.  ' 


In  the  procedure  just  described. 


column  averages  of  complete 


columns  are  used  as  scale  values.  Thus  if  all  columns  are  complete,  the 


entire  set  of  scale  values  may  be  readily  found  by  simply  using  column 
averages.  It  must  be  emphasized  that  when  elements  are  missing  from  a 
column,  it  is  not  correct  to  use  that  column  average  as  an  estimate  of  a 


scale  value,  even  for  an  approximation. 


I 


I 


APPENDIX  C. 


A  NUMERICAL  EXAMPLE 


Two  hundred  judges  were  asked  to  rank  four  types  of  equipment 
in  terms  of  loss  rate.  They  were  instructed  to  record  the  item 
(A,  B,  C,  and  D)  in  rank  order  according  to  their  best  judgement, 
but  to  omit  ranking  any  type  of  equipment  with  which  they  felt 
too  unfamiliar  to  make  a  judgement  about  its  loss  rate.  They 
were  also  instructed  that  if  they  felt  that  two  items  had  the 
same  loss  rate,  they  should  be  recorded  on  the  same  line. 

A  typical  judge's  response,  then,  would  be  a  completed  from 
as  shown  below,* 


Highest  Loss  Rate  - “ - 

A 

C 

Lowest  Loss  rate  D 

This  input  was  tallied  on  the  frequency  array  in  the  following  manner. 
Since  B  was  rated  at  the  top  of  this  judge's  ranking,  we  go  to  the  B 
column  of  the  frequency  array  and  in  that  column  we  record  tallies  in 
rows  corresponding  to  instances  (items)  that  this  judge  has  ranked 

‘Note  the  cues  given  to  the  first  and  last  rankings,  which  help  reduce 
the  chance  that  a  judge  will  list  instances  in  reverse  order. 
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below  B; 


The  completed  frequency  array  for  all  200  judges  is  shown  below. 


f.  . 

13 

A 

B 

C 

D 

A 

33 

1 

45 

B 

167 

30 

4 

118 

C 

! 

195 

166 

172 

D 

155 

82 

24 

Adding  across  the  diagonal,  we  see  that  not 
comparisons.  This  was  due  to  the  fact  that 
ttem  C  from  their  rankings. 

Continuing,  we  next  must  compute  the 


all  200  judges  made  all 
several  judges  omitted 


array  from  the  f^^ 


array 


by 


Because 
will  be 


f .  . 

13 


not  all  judges  ranked  all  items  ,  the  divisor  in  some  cases 
less  than  200.  The  p^j  array  is  computed  as 


13 


A 

B 

C 

D 


sums 


B 


D 


0.5  0.165  0.005  0.225 
0.835  0.5  0.153  0.590 
0.995  0.847  0.5  0.878 
0.775  0.410  0.122  0.5 


2.605  1.420  0.280  1.693 


From  the  p^^  array  we  see  that  in  comparing  C  and  A  the  judges  were  too 


close  to  unanimity,  and  thus  we  will  omit  this  comparison  (p^^  <  0.02, 
p  >  0.98)  in  the  z. .  array.  From  a  table  of  the  normal  distribution, 


Lne  array  is 


z .  . 

A 

B 

C 

D 

A 

0 

-0.974 

-0.755 

B 

0.974 

0 

-1.024 

0.227 

C 

1.024 

0 

1.170 

D 

0.755 

-0.227 

-1.170 

0 

Since  the  z^^  array  has  cells  which  are  empty,  we  cannot  use  the 
column  averages  to  compute  scale  values  for  all  the  items.  Instead,  we 
have  to  solve  simultaneous  equations  to  obtain  a  least-square  solution. 


Two  columns  of  the  z^^j  array  are 

complete , 

and 

two  are  not. 

by  computing  the  column  sums: 

j  A 

B 

C 

D 

Iz..  1.729 

-0.177  - 

2.194 

0.642 

•  i  1  j 

n .  3 

1 

4 

3 

4 

• 

Implicitly  setting  the  origin  of  the  scale  by  taking  the  average  scale 
value  as  zero,  we  immediately  compute  column  averages  for  the  two 
complete  columns  and  obtain  two  scale  values: 

S„  =  =  -0.044 

o  ■ 


=  0.161 

D  4— 

For  the  remaining  columns  (A  and  C) ,  the  set  of  equations  of  the 
form  of  (5)  is 

-  <=A  *  *  =D>  = 

^  =C  *  ®D>  = 


-2.194. 


we  may  write  the  two  equations  as 


and 


and  thus  we  have 


and 


2S^  =  1.729  +  0.117 
=  1.846 

2S^  =  -2.194  +  0.117 
=  -2.077 

=  0.923 

=  -1.039  . 


These  values  are  still  with  unspecified  unit  and  origin.  If 
we  estimate  that  the  loss  rate  for  C  is  10,  and  the  loss  rate  for 
item  D  is  30,  then  Equation  15  becomes 

L.  =  27.32  +  16.67Sj 

and  we  have,  finally. 

Item  Loss  Rate 


Hi  O  O 


APPENDIX  D 


RANKINGS  BY  JUDGES 


Judge : 

_1 

_2 

_3 

H 

_5 

_6 

_J_ 

_8 

10 

Highest 

46 

40 

40 

43 

41 

13 

13 

20 

46 

Chance 
of  Loss 

41 

41 

41 

10 

40 

14 

14 

23 

40 

42 

42 

42 

16 

42 

40 

49 

49 

26 

41 

40 

43 

19 

20 

43 

41 

43 

40 

29 

42 

10 

49 

14 

19 

13 

42 

41 

42 

23 

13 

16 

13 

13^ 

_14 

30 

30 

40 

43 

49 

43 

14 

30 

17 

35 

46 

42 

41 

43 

49 

16 

ilO 

46 

49 

10 

17 

14 

13 

30 

26 

42 

46 

20 

46 

11 

17 

30 

17 

1  43 

41 

29 

43 

30 

16 

19 

35 

23 

49 

40 

23 

29 

19 

30 

30 

14 

35 

16 

10 

16 

10 

35 

23 

17 

29 

17 

17 

17 

29 

10 

26 

23 

26 

20 

43 

14 

19 

23 

19 

35 

26 

20 

21 

49 

19 

14 

50 

23 

16 

19 

19 

29 

30 

20 

13 

26 

20 

20 

20 

46 

23 

35 

26 

21 

11 

26 

21 

29 

21 

35 

29 

21 

23 

20 

21 

21 

10 

26 

26 

35 

29 

50 

23 

35 

16 

46 

Lowest 
Chance 
of  Loss 

21 

49 

21 

50 

Delete: 

43 

23 

Orr.its ; 

11 

11 

11 

11 

11 

11 

17 

many 

11 

46 

13 

16 

50 

40 

50 

50 

50 

52 


Highest 

41 

41 

41 

13 

14 

42 

16,10 

40 

Chance 
of  Loss 

40 

40 

11 

14 

17 

.  41 

41 

4 

42 

42 

10 

49 

13 

43 

42 

3 

23 

23 

16 

43 

41 

14 

49 

46 

30 

30 

40 

40 

42 

10 

40 

1 

35 

35 

42 

17 

30 

13 

43 

1 

29 

29 

43 

42 

40 

30 

13 

1 

19 

19 

30 

19 

43 

35 

14 

2 

10 

10 

35 

10 

20 

40 

17 

1 

11 

11 

26 

16 

35 

17 

30 

1 

26 

26 

49 

46 

10 

29 

35 

2 

1 

13 

13 

23 

11 

29 

46 

19 

1 

1 

) 

14 

14 

46 

30 

26 

49 

23 

16 

16 

20 

29 

21 

13 

29 

17 

17 

29 

35 

49 

20 

26 

20 

20 

17 

23 

46 

19 

20 

21 

21 

19 

26 

16 

21 

21 

3 

13 

20 

23 

23 

46 

46 

46 

14 

21 

19 

26 

43 

43 

21 

50 

11 

50 

Lowest 
Chance 
of  Loss 

49 

49 

50 

50 

26 

Delete: 

13 

Omits : 

50 

50 

41 

11 

11 

1 

16 

50 

1 

2 

4 

5 

Highest 

13,14 

13 

13 

49 

Chance 

• 

of  Loss 

43,49 

49,17,14 

43 

13,14 

17 

40 

49 

17 

16,19 

41 

14. 

20,19,16,21 

40,41,42 

10,11 

40,41,42 

40 

29,30 

43 

10 

43 

10,11 

16 

16 

42 

20,26 

42 

17 

41 

21 

30,19,35 

29 

23 

26,23,29 

30 

10,23 

46 

20,46 

35 

26,46 

35 

21 

30,35 

50 

29 

26 

19 

Lowest 
Chance 
of  Loss 


20,21,23 
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